Ability to filter alerts by string parameters #92036

mikecote · 2021-02-19T16:03:18Z

Resolves #92010

In this PR, I'm allowing alert parameters to be filtered on string values.

elasticmachine · 2021-02-19T19:06:41Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

mikecote · 2021-02-19T20:50:08Z

src/core/server/saved_objects/service/lib/filter_utils.ts

+  const keys = key.split('.');
+  for (let i = 0; i < keys.length; i++) {
+    const path = `properties.${keys.slice(0, i + 1).join('.properties.')}`;
+    if (get(indexMappings, path)?.type === 'flattened') {
+      return true;
+    }
+  }


@elastic/kibana-core this PR is ready for review. I had to add this piece of code and a few early returns to support flattened type fields in alerting's use case.

pmuellr

LGTM!

I'm curious now if this can solve our "internal tags" requirement. I think it will require the per-type alert UIs to pre-fill these some new "hidden" params (hidden from the UI). For some reason I didn't realize you could access the individual fields in a flattened type in a query, I thought it just jammed all the text for all keys into a single field (which makes me curious how it's implemented)!

gmmorris · 2021-02-22T09:54:45Z

x-pack/test/alerting_api_integration/spaces_only/tests/alerting/find.ts

+      const response = await supertest.get(
+        `${getUrlPrefix(
+          Spaces.space1.id
+        )}/api/alerts/_find?filter=alert.attributes.params.strValue:"my b"`


It's feeling like we're leaking Alerting internal implementation here...

I'm ok with that, as it feels like the overhead of trying to abstract the attributes away is probably not worth it.... but I. would like this to have explicit documentation added (asciidocs/dev docs) as I don't think it's intuitive :/

@gmmorris Agreed, I always felt like that with the KQL filter. Though, the enforcement of attributes is made by the saved objects service so I figured we'd pass it through as we do with other fields.

Under filter docs (https://www.elastic.co/guide/en/kibana/master/alerts-api-find.html) the params would work the same as any other field but I think it makes sense to document that params are analyzed as flattened and point to those docs so users understand the limitations of these fields.

Thoughts?

Yeah, makes sense.
Putting myself in the mindset of a developer using our APIs - if the API explains what params is and leads me to an API that explains how to query against it - that's fine.

Do we actually explain that the Alerts are stored as SOs? 🤔
I'm not sure the SO Api makes it clear how to query attributes though 🤔 I'll take a look.

From the docs:

(Optional, string) A KQL string that you filter with an attribute from your saved object. It should look like savedObjectType.attributes.title: “myTitle”. However, If you used a direct attribute of a saved object, such as updatedAt, you will have to define your filter, for example, savedObjectType.updatedAt > 2018-12-22.

I think the only extra mention we need is that they're flattened and have filtering limitations.

gmmorris

LGTM, I'd just like this to add some docs as I don't think using this is intuitive considering it relies on Alerting/SOs internal implementation

gmmorris · 2021-02-22T09:55:45Z

@elasticmachine merge upstream

mikecote · 2021-02-22T12:18:05Z

@pmuellr

I'm curious now if this can solve our "internal tags" requirement. I think it will require the per-type alert UIs to pre-fill these some new "hidden" params (hidden from the UI). For some reason I didn't realize you could access the individual fields in a flattened type in a query, I thought it just jammed all the text for all keys into a single field (which makes me curious how it's implemented)!

Yeah, it works pretty well which is why I didn't understand why there would be a use case for "internal tags" if we had this. I think we'll be able to close the issue in favour of this and ask for use-cases if ever this doesn't solve it. (one less thing to maintain).

mikecote · 2021-02-22T13:55:56Z

docs/api/alerts/find.asciidoc

@@ -49,6 +49,8 @@ Retrieve a paginated set of alerts based on condition.
 NOTE: As alerts change in {kib}, the results on each page of the response also
 change. Use the find API for traditional paginated results, but avoid using it to export large amounts of data.

+NOTE: Alert `params` are stored as {ref}/flattened.html[flattened] and analyzed as `keyword`.


@gchaps let me know what you think, I wanted to state that filtering on the params field has limitations.

I've merged this PR, but I will create a follow-up if you would like changes to this line.

kibanamachine · 2021-02-22T15:56:59Z

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`alert`	32	33	+1

History

💚 Build #108300 succeeded 6a47f1c
💚 Build #108061 succeeded 9c1b890

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mikecote

* Initial commit * Update comment * Return early * Add docs Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-02-22T16:00:30Z

💚 Backport successful

✅ 7.x / #92219

Successful backport PRs will be merged automatically after passing CI.

* master: Ability to filter alerts by string parameters (elastic#92036) [APM] Fix for flaky correlations API test (elastic#91673) (elastic#92094) [Enterprise Search] Migrate shared role mapping components (elastic#91723) [file_upload] move ml Importer classes to file_upload plugin (elastic#91559) [Discover] Always show the "hide missing fields" toggle (elastic#91889) v2 migrations should exit process on corrupt saved object document (elastic#91465) [ML] Data Frame Analytics exploration page: filters improvements (elastic#91748) [ML] Data Frame Analytics: Improved error handling for scatterplot matrix. (elastic#91993) [coverage] speed up merging results of functional tests (elastic#92111) Adds a Reason indicator to the onClose handler in AddAlert and EditAlert (elastic#92149)

* Initial commit * Update comment * Return early * Add docs Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>

Initial commit

72fb3f5

mikecote added release_note:enhancement Feature:Alerting v8.0.0 Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.13.0 auto-backport Deprecated - use backport:version if exact versions are needed labels Feb 19, 2021

mikecote self-assigned this Feb 19, 2021

mikecote added 2 commits February 19, 2021 11:07

Update comment

461016b

Return early

9c1b890

mikecote marked this pull request as ready for review February 19, 2021 19:06

mikecote requested review from a team as code owners February 19, 2021 19:06

mikecote commented Feb 19, 2021

View reviewed changes

pmuellr approved these changes Feb 19, 2021

View reviewed changes

pgayvallet approved these changes Feb 22, 2021

View reviewed changes

gmmorris reviewed Feb 22, 2021

View reviewed changes

gmmorris approved these changes Feb 22, 2021

View reviewed changes

Merge branch 'master' into alerting/params-as-flattened

6a47f1c

Add docs

5e16f06

mikecote commented Feb 22, 2021

View reviewed changes

mikecote requested a review from gchaps February 22, 2021 13:56

mikecote merged commit 0c2495a into elastic:master Feb 22, 2021

kibanamachine mentioned this pull request Feb 22, 2021

[7.x] Ability to filter alerts by string parameters (#92036) #92219

Merged

mikecote mentioned this pull request Feb 22, 2021

Request for alerting internal tags structure #58417

Closed

mikecote mentioned this pull request Feb 22, 2021

Update docs on filtering alerts by string params #92239

Merged

This was referenced Feb 23, 2021

[Security Solution][Detections] Enable filtering of string fields within Rule Management #92333

Open

[Security Solution][Detections] Investigate removing usage of internal tags #92334

Open

pmuellr mentioned this pull request Feb 23, 2021

[Alerting] Add a tie breaker field to alerts #62002

Closed

sorenlouv mentioned this pull request Mar 19, 2021

Make alert params searchable #50213

Open

pmuellr mentioned this pull request May 25, 2021

[alerting] migrating an alert with a large alert param into 7.13.0 will cause a migration error #100607

Closed

mikecote mentioned this pull request Jul 7, 2021

Add migration test for rule params flattened type #104659

Open

pmuellr mentioned this pull request Sep 27, 2021

[saved objects] add support for runtime fields in Saved Object find() requests #113152

Open

spong mentioned this pull request Feb 7, 2022

[Security Solution][Detections] Remove usage of __internal_ tags from Security Detection Rules #124899

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to filter alerts by string parameters #92036

Ability to filter alerts by string parameters #92036

mikecote commented Feb 19, 2021 •

edited

Loading

elasticmachine commented Feb 19, 2021

mikecote Feb 19, 2021

pmuellr left a comment

gmmorris Feb 22, 2021

mikecote Feb 22, 2021

gmmorris Feb 22, 2021

mikecote Feb 22, 2021

gmmorris left a comment

gmmorris commented Feb 22, 2021

mikecote commented Feb 22, 2021

mikecote Feb 22, 2021

mikecote Feb 22, 2021

kibanamachine commented Feb 22, 2021

kibanamachine commented Feb 22, 2021

Ability to filter alerts by string parameters #92036

Ability to filter alerts by string parameters #92036

Conversation

mikecote commented Feb 19, 2021 • edited Loading

elasticmachine commented Feb 19, 2021

mikecote Feb 19, 2021

Choose a reason for hiding this comment

pmuellr left a comment

Choose a reason for hiding this comment

gmmorris Feb 22, 2021

Choose a reason for hiding this comment

mikecote Feb 22, 2021

Choose a reason for hiding this comment

gmmorris Feb 22, 2021

Choose a reason for hiding this comment

mikecote Feb 22, 2021

Choose a reason for hiding this comment

gmmorris left a comment

Choose a reason for hiding this comment

gmmorris commented Feb 22, 2021

mikecote commented Feb 22, 2021

mikecote Feb 22, 2021

Choose a reason for hiding this comment

mikecote Feb 22, 2021

Choose a reason for hiding this comment

kibanamachine commented Feb 22, 2021

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

History

kibanamachine commented Feb 22, 2021

💚 Backport successful

mikecote commented Feb 19, 2021 •

edited

Loading